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•  Overall  Goal:  understand  the  unique  cognitive  mechanisms  that  should  be  employed  to 
optimize  collaborative  decision-making  activity  in  a  geographically  distributed  and 
time-delayed  situation 

•  Objectives:  (1)  to  understand  the  cognitive  process  of  building  knowledge  in  an  asynchronous, 

distributed  collaboration  environment 


(2)  to  develop  an  empirically-based  theory  of  collaboration,  including  knowledge 
building  components,  during  asynchronous,  distributed  collaboration 


(3)  to  understand  how  agents  can  support  humans  in  achieving  collaborative 
knowledge  during  asynchronous,  distributed  collaborative  problem  solving 

•  FY  04  Objectives: 

(1)  Conduct  experiments  in  collaborative  problem  solving  in  both  face-to-face  and 
asynchronous,  distributed  environments  to  understand  the  unique  cognitive 
processes  within  asynchronous,  distributed  collaboration  (emphasis  on 
Collaboration  Mode  &  Knowledge  Uncertainty) 

(2)  Update  model  of  collaboration  based  on  experimental  results 


TECHNICAL  PLAN 
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Science  Concept  Researched: 


•  What  is  the  knowledge  building  process  humans  use  in  asynchronous,  distributed 
collaborative  problem  solving? 


•  Understand  the  effect  of  the  following  variables  on  the  knowledge  building  process 
during  asynchronous,  distributed  collaboration? 

*  Collaboration  Modes  (  face-to-face  versus  asynchronous,  distributed) 

*  Knowledge  Distribution  (homogeneous  versus  heterogeneous) 

*  Knowledge  Uncertainty  (static  knowledge  versus  dynamic  knowledge) 


•  What  forms  of  agent  support  facilitate  knowledge  building  in  asynchronous, 
distributed  collaborative  problem  solving? 


Empirical  Model 
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•  Collaboration  Task: 

>  Expanded  NEO  Mission  Scenario  (Warner,  Wroblewski,  Shuck,  Cowen,  Letsky,  2003) 

*  developed  with  expertise  from  operational  personnel  (Navy  Seal,  Marine,  Army  aircrew) 

>  Mission  Statement  — 

The  time  is  2:00am,  January  15.  Your  mission  is  to  rescue  3  stranded  Red  Cross  workers  from  a  church  basement,  on 
a  remote  island,  caught  in  the  middle  of  guerilla  warfare,  within  24  hours.  The  situation  is  described  in  the  next  few 
pages  along  with  the  assets  of  US  forces  which  are  available  to  rescue  the  workers.  You  need  to  work  together  and 
develop  a  course  of  action  (using  ANY  assets  available  to  you),  which  includes  a  plan  for  getting  to  the  church,  a  plan 
for  evacuating  the  workers,  and  a  plan  for  the  return  to  the  Army  base  or  aircraft  carrier.  The  course  of  action  solution 
can  be  an  Army,  Marine,  Navy  Seals  solution,  or  a  combination  of  the  assets  of  the  three.  You  want  to  choose  the 
optimal  and  most  efficient  solution.  You  want  to  minimize  damage  to  the  village  and  villagers;  you  want  to  avoid  contact 
with  enemy  if  possible,  and  to  rescue  the  workers  safely.  However,  the  rules  of  engagement  are  that  any  forces  will 
defend  themselves  if  needed.  Good  Luck! 

•  Asynchronous,  Distributed  Collaboration  Environment: 


>  Modified  Ewall  workstation  and  visualization  area 
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-  Collaboration  Mode  (face-to-face  vs  asynchronous,  distributed) 

*  Face-to-Face  =  team  interacts  synchronously  with  each  other  through  speech 

*  Asynchronous,  Distributed  =  team  interacts  with  each  other  at  different  times 
and  from  different  locations  through  the  Ewall  collaboration  environment 


-  Knowledge  Uncertainty  (static  vs  dynamic  knowledge) 

*  static  knowledge  =  all  information  (I.e.  background  information,  rebels,  Navy 

Seals,  Marine  and  Army  assets,  Intell,  weapons  and 
environment  expertise  information)  remains  the  same 
throughout  the  collaborative  NEO  scenario  problem 


*  dynamic  knowledge  =  Selected  information  (I.e.  rebel  location  and  weather) 

changes  at  a  standard  time  in  the  collaborative  NEO 
scenario  problem 
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CASC  PHASE  II  EXPERIMENTAL  DESIGN 
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Phase  II 

•  32  groups  total 

•  3  subjects  /  group 

•  96  subjects  total 


DEPENDENT  VARIABLES: 

-  Ewall  cards  and  face-to-face  audio  /  video  recordings  including  time  stamp  per  response 
(I.e.  cards  and  speech) 

-Quality  of  Decision  (scoring  of  the  team’s  final  plan  against  final  plan  developed  by  an 
operational  team  consisting  of  Marine,  Army, and  SEAL  personnel) 

-  Total  time  to  successfully  complete  the  problem-solving  task  (time  from  the  beginning  of 
the  task  until  task  completion) 

-  Collaboration  Maps  (post  session  -  subjects  construct  a  map  of  their  view  of  the  stages  & 
cognitive  process  states  of  team  collaboration) 

-  Subjective  Questionnaire  -  measuring  expertise,  trust  between  team  members,  and  general 
collaboration  opinions  among  members 


TYPES  OF  DATA  ANALYSES 

Phase  I,  and  II  Experiments 
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•  Verbal  Protocol  Communication  Analyses-  identification  of  collaboration  stages  and  cognitive 
process  states  compared  across  collaboration  mode  and  knowledge  uncertainty.  Compare  results 
to  model  of  collaboration. 


•  Transition  State  Diagrams  -  representation  of  the  dynamic  team  collaborative  behavior  between 
collaboration  stages  and  between  cognitive  process  states  within  each  collaboration  stage  compared 
across  collaboration  mode  and  knowledge  uncertainty.  Compare  results  to  model  of  collaboration. 


•  Parametric  statistics  for  analyzing  time,  and  frequency  within  each  collaboration  stage  and 

cognitive  process  state  across  collaboration  mode  and  knowledge  uncertainty  conditions.  Also  used 
for  analyzing  quality  of  decision,  total  time  to  complete  task  and  questionnaire  data. 


•  Collaboration  Maps  -  determine  the  degree  of  convergence  between  individual  mental  model  Ss 
regarding  collaboration  stages  and  cognitive  processes.  In  addition,  compare  how  an  individual 
thinks  a  group  makes  a  decision  in  a  collaborative  setting  and  how  the  group  actually  performs. 
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•  Member  National  &  International 
Research  Panels 

•  Recent  efforts :  CASC,  Agent 
Learning,  ADSS,  ANGEL,  SCC 

•Tools:  local  web  server,  Pathfinder, 
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Experience  X  Capabilities 

•  Over  25  Years  Experience  in 
Decision  Making  /  Automation  Research 


**  Produced  by  MIT  under  ONR  CKM  program 


•  Joint  efforts  (e.g.  NAVAIR  TSD,  JFCO 
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Model  Evolution 
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ollaborative  work  /  capability  to  achieve  goals 


KNOWLEDGE  BUILDING  PROCESS 

CKSSE.srRANSr<>H'l'T'VE  (Stahl,  2000)  KNO>VALEGDLGE  'FORM 


C  ollaborative 


|p 


i 


Formalize  &  objectify 

4 

Use  in  performing  task 

▼ 

underspending 


shared 

understanding 
C  la  rifyA  meanings 


other  people’s 


M  ETRICS 

C  ollaboration 

•  information  rich  group  discussions 

•  collective  decision  making 

•  integrated,  joint  solutions  derived 
from  group 

Knowledge  Building  Process 

•  Thinking  Aloud  Protocol  (verbalize) 

•  Concept  Map  (pre  /  post  session) 

•  Percentage  of  Domain  Relevant 
Information  exchanged  (verbal  /  text 
protocol  analyses) 

•  Number  of  Integrated,  Joint  Solutions 

•  Time  to  successfully  complete 

common  task _ 


IDE 


STRUCTURAL  MODEL  OF  TEAM  COLLABORATION 


Problem  Area 
Characteristics 


•  culturally  diverse 

•  heterogeneous  knowledge 

•  unique  roles 

•  command  structure 
(hierarchical  vs.  flat) 

•  rotating  team  members 


•  team  decision  making,  COA 
selection 


Meta-Cognitive: 

•  iniiv  irtiial  understanding  of  •  goal  deselopmau 

pmhlan  con.tn.ons  .  team  cnx1cl  of  problem 

•  indivutial  moual  model  development .  te)m  t0  snlvt 
of  situational  significance 


uris  mental  model  changes  •  conpwre  pioblem  solution 


*  problem  identification 


•esttjbfeh 

•  establish  data  filtering  methods 

•  establish  meaning  transfer  i 


•  goal  definition 

•  iterative  information  collection 
&  analysis 


•Knowledge  Building: 

•  individual  task  knowledge 

•  individual  team  knowledge 


team  mental  model  of  team 
team  task  knowledge 


shared  understanding 
collaborative  knowledge 


•  collaborative  knowledge 

•  shared  understanding 


•  tjjal  requirements 

•  exit  criteria 


n  for  Information  Processing  and  Knowledge  Building  (applies  to  all  stages): 


•  presenting  individual  information 


•  discussing  individual  information 

•  negotiating  perspectives 

•  discussion  of  possible  solutions 


an  generated  information 
•  providing  rationale  for  individual  solutions 


Preliminary  Conceptual  Model  of  Collaboration 


(Stages) 


Level  1  /  Team  Orientatiob, 

•  getting  acquainted 

•  clarifying  task 
initial  attitudes 


_  Level  2 

(Processes) 


Individual 

Understanding 


at^to  T 


Accumulation 
of  facts  (Team) 


Select 
&  Disculs 


|  |  =  Knowledge 

Building 


•  Decision  Alternatives 

•  Criticism  of  Alternatives 

(with  rationale) 


Iteration  loop  for  selecting  dccisionT  alternatives 


Collaborative 

Knowledge 


Team  Shared 
Understanding 


Team 

Consensus 


Implement 
Decision  To  Solve 
Task 


Problem  Area 
Characteristics 


MODEL  OF  TEAM  COLLABORATION 
(MACROCOGNITIVE  PROCESS  FOCUS) 


•  asynchronous 

•  distributed 

•  culturally  diverse 

•  heterogeneous  knowledge 

•  unique  roles 

•  command  structure 
(hierarchical  vs.  flat) 

•  rotating  team  members 


•  team  decision  making,  COA 
selection 

•  develop  shared  understandi 

•  intelligence  analysis 
(team  data  processing) 


Collaboration  Stages  &  Cognitive  Processes 


Meta-Cognitive. 

*  individual  conversion  of 
data  to  knowledge 


m  integration  of  individual 
knowledge  for  common  understanding 


solution 


Macro-Cognitive: 

•  individual  mental  model 
construction 

•  knowledge  interoperability 
development 

•  individual  task,  team  and  domain 
knowledge  development 

•  individual  knowledge  object 
development 

•  individual  visualization  and 
representation  of  meaning 


•  knowledge  interoperability 
development 

•  iterative  information  collection 

and  analysis 

•  team  shared  understanding 
development 

•  develop,  rationalize,  &  visualize 
solution  alternatives 

•  convergence  of  individual  mental 
models  to  team  mental  model 


•  team  negotiation  of  solution 
alternatives 

•  team  pattern  recognition 

•  team  shared  understanding 
development 

•  convergence  of  individual 
mental  models 

•  critical  thinking 

•  sharing  hidden  knowledge 


•  solution  adjustment  to  i  1 
fit  goals  and  exit  criteria  j 


•  compare  problem  solution 
against  goals 

•  team  shared  understanding 
development 

•  convergence  of  individual 
mental  models  of  solution 

•  analyze,  revise  output 


_l 


Mechanisms  for  achieving  Meta,  Macro,  and  Micro-Cognitive  Processes  (applies  to  all  stages) 

•  Verbal  communications  :  presenting  and  discussing  individual  information,  discussing  tea  m  generated  information.  1  O 
questioning,  agreeing  /  disagreeing,  negotiating  perspectives  ,  discussing  possible  solutions,  providing  rationale.  X  O 

•  Non-Verbal  communications  :  facial  expressions,  voice  clues  (vocal  paralanguage),  hand  gestu  res,  body  movements  (kinesics), 
touch  (haptics),  personal  space,  drawing,  text  messages,  augmented  video,  affordances  (cognition  in  objects). 


MODEL  OF  TEAM  COLLABORATION 

(MACRO  -  COGNITIVE  PROCESS  FOCUS) 


NAV^AIR 


Problem  Area 
Characteristics 


Collaborative  Situation 

Parameters: 

•  time  pressure 

•  information/knowledge 
uncertainty 

•  dynamic  information 

•  large  amount  of  knowledge 
(cognitive  overload) 

•  human-agent  interface 
complexity 

Team  Tvoes 

•  asynchronous 

•  distributed 

•  culturally  diverse 

•  heterogeneous  knowledge 

•  unique  roles 

•  command  structure 
(hierarchical  vs.  flat) 

•  rotating  team  members 

Operational  Tasks 

•  team  decision  making,  COA 
selection 

•  develop  shared  understands 

•  intelligence  analysis 
(team  data  processing) 


Collaboration  Stages  &  Cognitive  Processes 


i - 

|  Meta-Cognitive: 

]  •  individual  conversion  of 
i  data  to  knowledge 


•  team  integration  of  individual 
knowledge  for  common  understanding 


1  team  agreement  on  a  common  j 
solution 


■  solution  adjustment  to 
fit  goals  and  exit  criteria 


r 

I  Macro-Cognitive: 

|  •  individual  mental  model 
I  construction 
I  •  knowledge  interoperability 
I  development 

I  •  individual  task,  team  and  domain 

■  knowledge  development 

I  •  individual  knowledge  object 
•  development 

■  •  individual  visualization  and 

■  representation  of  meaning 


knowledge  interoperability 
development 

iterative  information  collection 
and  analysis 

team  shared  understanding 
development 

develop,  rationalize,  &  visualize 
solution  alternatives 
convergence  of  individual  mental 
models  to  team  mental  model 

individual  task,  team  and  domain 
knowledge  development 


team  negotiation  of  solution  j 

alternatives  5 

team  pattern  recognition 
team  shared  understanding  j 

development  ? 

convergence  of  individual  j 

mental  models  ; 

critical  thinking  ; 

sharing  hidden  knowledge  j 

individual  task  knowledge  J 

development  . 

team  task  knowledge 
development  : 

i 

_ 1 


•  compare  problem  solution 
against  goals 

•  team  shared  understanding 
development 

•  convergence  of  individual 
mental  models  of  solution 

•  analyze,  revise  output 


Mechanisms  for  achieving  Meta.  Macro,  and  Micro-Cognitive  Processes  (applies  to  all  stages) 

•  Verbal  communications:  presenting  and  discussing  individual  information,  discussing  tean  generated  information, 
questioning,  agreeing  /  disagreeing,  negotiating  perspectives  discussing  possible  solutions,  providing  rationale. 

•  Non-Verbal  communications:  facial  expressions,  voice  clues  (vocal  paralanguage),  hand  gestires,  body  movementsl(9inesics), 
touch  (haptics),  personal  space,  drawing,  text  messages,  augmented  video, affordances  (cognition  in  objects). 


CASC  Phase  II  Results 
(verbal  protocol  analyses) 
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Total  Utterance  Frequencies  by  Collaboration  Stages 

&  Collaboration  Mode 


Knowledge  Construction  Team  Problem  Solving 

CM:  F  =  60.81317,  p  =  0.00  CM:  F  =  55.27290,  p  =  0.00 

KU:  F  =  0.22663,  p  =  0.637727  KU:  F  =  1.66238,  p  =  0.207833 
CM*KU:  F  =  0.33155,  p  =  0.569348CM*KU:  F  =  2.27126,  p=  0.142993 
N  =  32  (Teams)  N  =  32  (Teams) 


Team  Consensus  Outcome  Evaluation  &  Revision 

CM:  F  =  26.87553,  p  =  0.000017  CM:  F  =  8.12793,  p  =  0.008094 


KU:  F  =  3.75526,  p  =  0.062786 
CM*KU:  F  =  3.25928,  p  = 
0.081785 
N  =  32  ITeamsI 


KU:  F  =  0.09557,  p  =  0.759504 
CM*KU:  F  =  0.001 18,  p  = 
0.972843 
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Collaboration  Mode 


□  Face  to  Face 
■  Asynchronous 
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O  utcome 
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Mean  %  of  Time 
Spent  in  Stage 


CASC  Phase  II  Results 
(verbal  protocol  analyses) 


NAV^AIR 


Mean  %  of  Time  by  Collaboration  Stages 
&  Collaboration  Mode 


Know  ledge  Construction: 
CM:  F  =  6.5050,  p  =  0.016512 
KU:  F  =  0.1540,  p  =  0.697755 
CM*KU:  F  =  0.00,  p  =  0.995635 
N  =  32  (Teams) 


Team  Problem  Solving: 

CM:  F  =  5.601,  p  =  0.025104 
KU:  F  =  0.745,  p  =  0.395359 
CM*KU:  F  =  0.00,  p  =  0.990072 
N  =  32  (Teams) 


Team  Consensus: 

CM:  F  =  0.81404,  p  =  0.374624 
KU:  F=  1.56708,  p  =  0.220989 
CM*KU:  F  =  0.04719,  p  =  0.829603 
N  =  32  (Teams) 


Outcome,  Evaluation  &  Revision: 
CM:  F  =  0.08520,  p  =  0.772517 
KU:  F  =  0.23715,  p  =  0.630062 
CM*KU:  F  =  0.41881,  p  =  0.522804 
N  =  32  (Teams) 
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□  Face  to  Face 
B  Asynchronous 


u 


L 


H 


Knowledge 

Construction 


Team 

Problem  Solving 


Team 

Consensus 


Outcome 
Evaluation  & 
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Collaboration  Stages 
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CASC  Phase  II  Results 
(verbal  protocol  analyses) 
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Mean  %  of  Time  Spent  in  Significant 
Cognitive  Processes  by  Knowledge  Construction 


MCimm: 

Individual  Task 
Knov\tedae  Development 
CM:  F  =4.9790,  p= 0.033841 
Kit  F= 0.0486,  p= 0.827031 
CMKU:  F= 0.2603,  p= 0.613895 
N= 32  (teams) 


Mcvrm: 

Individual  Visualization 
&  Representation  of  Meaning 

CM:  F  =  13.37226,  p  =  0.001046 
KQ  F= 0.07915,  p  =  0.780524 
CMKQ  F  =  0.25495,  p= 0.617562 
N  =  32  (teams) 


Mccmm: 

Convergence  of  Individual  Mental 

Models  to  Team  Mental  Model 

CM:  F  =  12.05386,  p=0.0016'B 
KUt  F  =  0.35939,  p= 0.55366731 
CM*KU  F  =  130053,  p= 0.263778 
N= 32  (teams) 
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Knowledge  Development  Representation  of  Meaning  Models  to  Team  Mental  Model 

Significant  Cognitive  Processes 


Mean  %  of  Time  Spent  in 
Significant  Cognitive  Processes 


CASC  Phase  II  Results 
(verbal  protocol  analyses) 


NAV^AIR 


Mean  %  of  Time  Spent  in  Significant 
Cognitive  Processes  by  Team  Problem  Solving 


Me  vrm: 

Individual  Visualization 
&  Representation  of  Meaning 

CM:  F  =  6.83871,  p  =  0.071206 
KU:  F  =  0.21S10,  p  =  0.645619 
CMKU:  F  = 0.74289,  p= 0.396057 
N  =  32  (teams) 


Mckio: 

Knowledge 

Interoperability  Development 

CM:  F  =  14.1314,  p  =  0.00078 
KU:  F  =  0.0630,  p  =  0.803688 
CMKU:  F  =  TB37,  p  =  0.285872 
N  =  32  (teams) 


Mccmm: 

Convergence  of  Individual 
Mental  Models  to  Team  Mental  Model 

CM:  F  =4.06083,  p  =  0.53581 
KU:  F  =4.52292,  p  =  0.042392 
CMKU:  F= 0.00929,  p  =  0.923090 
N  =  32  (teams) 
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CASC  Phase  II  Results 
(verbal  protocol  analyses) 
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Mean  %  of  Time  Spent  in  Significant 
Cognitive  Processes  by  Team  Consensus 

MC  cm  m : 

Convergence  of  Individual 
Mental  Models  to  Team  Mental  Models 
CM:  F  =  4.9790,  p  =  0.033841 
KU:  F  =  0.0486,  p  =  0.827031 
CM*KU:  F  =  0.2603,  p  =  0.613895 
N  =  32  (teams) 
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Probability  of  Occurrence: 
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Probability  of  Occurrence: 
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Probability  of  Occurrence: 
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Outcome, 
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Probability  of  Occurrence: 

F2F:  0%  AD:  1% 
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Asynchronous, 
Distributed  (AD) 


CASC  Phase  II  Conclusions 
(outcome  measures) 
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+  There  were  no  significant  difference  in  time  to  complete  NEO  scenario  between  face- 
to-face  and  asynchronous ,  distributed  teams 

*  these  results  differ  from  the  literature  with  asynchronous,  distributed  teams 
taking  longer  to  complete  tasks  than  face-to-face  teams 

*  difference  from  current  literature  could  be  due  to  realistic  complexity  of  NEO 
scenario  task  along  with  the  60  minute  time  constraint  (I.e.  time  pressure) 

to  solve  problem 

+  The  face-to-face  and  asynchronous ,  distributed  teams  achieved  the  same  high  quality 
solution  in  the  same  amount  of  time  but  the  asynchronous,  distributed  teams 
required  less  communication 

*  results  suggest  that  the  structure  of  the  Ewall  environment  (I.e.  the  cards 
coupled  with  the  modules)  permits  more  effective  communication  and 
collaboration  compared  to  the  face-to-face  team  environment 

*  results  suggest  that  the  Ewall  asynchronous,  distributed  teams  could  focus  on 
the  problem  better  and  communicate  more  relevant  information  and  knowledge 
between  team  members. 


CASC  Phase  II  Conclusions 

Collaboration  Stages 


NAV^AIR 


■  Results  support  that  both  face-to-face  and  asynchronous,  distributed  teams  go 
through  the  four  collaboration  stages,  Knowledge  Construction,  Team  Problem 
Solving,  Team  Consensus  and  Outcome,  Evaluation  and  Revision 

*  Asynchronous,  distributed  teams  made  significantly  fewer  utterances  in  the  four 
collaboration  stages  compared  to  the  face-to-face  teams  due  to  the  structure  of  the 
Ewall  environment 

*  Face-to-face  teams  spent  significantly  more  time  in  knowledge  construction  while 
the  asynchronous,  distributed  teams  spent  significantly  more  time  in  team  problem 
solving.  Data  suggest  that  Ewall  helped  focus  team  problem  solving  behavior 
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CASC  Phase  II  Conclusions 

Mean  Percent  of  Time 

Spent  in  Significant  Cognitive  Process  States 


NAV^AIR 


I  Knowledge  Construction 

*  During  knowledge  construction  the  face-to-face  teams  spent  significantly 

more  time  in  individual  task  knowledge  development,  individual  visualization  & 
representation  of  meaning,  and  convergence  of  individual  mental  model  to  team  mental 
model  compared  to  asynchronous,  distributed  teams 

■  Team  Problem  Solving 

*  During  team  problem  solving  the  face-to-face  teams  spent  more  time  in  individual 
visualization  &  representation  of  meaning  compared  to  asynchronous,  distributed  teams 

*  Asynchronous,  distributed  teams  spent  more  time  conducting  knowledge  interoperability 
between  team  members  than  face-to-face  teams 

*  Dynamic  teams  spent  more  time  in  converging  individual  mental  models  to  the  team  mental 

model  compared  to  static  teams. 

■  Team  Consensus 

*  During  team  consensus  there  is  a  significant  interaction  with  convergence  of  individual 
mental  models  to  team  mental  model  between  collaboration  mode  and  knowledge  uncertainty. 
Face-to-face,  dynamic  teams  spent  the  most  amount  of  time  followed  by  asynchronoj^, 
distributed,  static;  face-to-face  static  and  then  asynchronous,  distributed  dynamic. 


Summary  Conclusions  for  Team  Collaboration  Model 

CASC  Phase  I  and  II  Experiments 

Collaboration  Stages 

Cognitive  Process  Knowledge  Team  Problem  Team 

States  Construction  Solving  Consensus 


NAV^AIR 


Outcome,  Evaluation  & 
Revision 


individual  mental  model  construction 

individual  task  knowledge  development 

visualization  &  representation  of  meaning 

knowledge  interoperability 

iterative  information  collection 

team  shared  understanding 

develop  solution 

alternatives 

convergence  of  mental  models 

team  agreement  on  a  solution 

team  negotiation  of  solution  alternative 

team  pattern  recognition 

✓ 

CASC  Team  Collaboration  Model 

Knowledge  Construction  Representation 


NAV^AIR 


Cognitive  Process  States 

Knowledge 

Construction 

individual  mental  model  construction 

✓ 

individual  task  knowledge  development 

S 

visualization  &  representation  of  meaning 

S 

knowledge  interoperability 

s 

iterative  information  collection 

team  shared  understanding 

develop  solution 

alternatives 

convergence  of  mental  models 

► 


Note:  Representation  for  Team  Problem  Solving 
And  Team  Consensus  TBD  in  FY05 
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ForceNet  Transition 


Network  Centric  Information  Grid 


J  STARS 


SLCM 


NEWS  MODULE 


DATABASE  MODULE 


EXCHANGE  MODUL£ 


Exchange 

Server 


DA,  VA 


Asynchronous,  Distributed 
>~Ewall  Collaboration  Tool 
With 

Agent  Based 
Collaboration  Support 


Database 

View 


Exchange 

View 
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Special  Warfare  Transition 


NAV^AIR 


Asynchronous ,  Distributed  Ewall  Collaboration  Tool  With  Agent  Based  Collaboration  Support 


V 

•  More  timely  and  accurate 
mission  decisions 
(C2  down  to  individual 
warfighter) 


V 

•  Improved  intelligence, 
mission  planning  and 
execution 
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NAVAIR  Testbed 

Ewall  Demonstration  with  Special  Warfare  Scenario 
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E-Wall  as  a  Collaboration  Tool 
for  Intelligence  Analysis,  and  Mission  Planning 
for  Special  Operations  Forces 


The  Hunt  For  Saddam  Hussein 
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Hunt  For  Saddam 
Background  Information 


R 


•  Complex  and  difficult  mission 

-  Many  sympathizers,  family,  friends 

-  Extensive  preparations  for  personal  survival 

•  Body  doubles,  numerous  palaces,  large  cash  reserves  and  foreign 
bank  accounts 

•  Previous  attempts  on  life  created  very  effective  survival  techniques 

-  Y et,  large  disaffected  population  and  ethnic  groups 

•  Many  hated  regime  and  see  his  demise  as  a  positive  development 

•  Requires  massive  intelligence  effort 

-  Fusion  of  hundreds  of  analysts,  nodes,  feeds 

-  Networks  across  various  nations,  agencies,  services, 
commands 
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Illustrative  SOF  Planning  Cycle 


•  Intel  (J-2)  feeds  system 

-  Analyze  and  assess  intelligence  “take”  (fusion) 

•  All  source,  all  types,  many  “feeds” 

•  When  Intel  becomes  actionable,  provide  target  locations 

-  J-2  pass  info  to  Planners  (J-5) 

•  Planners  conduct  mission  feasibility  analysis 

•  Unit  level  personnel  plan  tactics,  Hq.  personnel  coordinate  platforms 
and  other  unit  support 

•  Briefback  to  commander,  if  approved,  execute  mission 

•  Operations  (J-3)  executes  and  monitors  mission 

-  After  mission,  intelligence  take  feeds  process  again 
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Intelligence  Categories 


•  HUMINT  (Human) 

-  Iraqi  Sources,  “Walk  in”  information 

-  Detainee  interrogations;  Document  exploitation  (DOCEX) 

-  Cooperative  governments;  Overseas  sources 

•  IMINT  (Imagery) 

-  Satellite  Imagery 

-  UAV  Imagery 

-  Ground  tactical  imagery 

•  SIGINT  (Signals) 

-  Telephone  intercepts 

-  Radio  intercepts 

•  OSINT  (Open  Source) 

-  Foreign  Broadcast  Information  Service 

-  News  reports 

-  News  Channels 

•  MASINT  (Measurements  and  Signals) 

-  Thermal  infrared  heat  imaging 

-  Effluent/debris  collection 
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Planning  Elements 


•  Intelligence 

-  Location  (s) 

-  Threat  information 

-  Movement  patterns 

•  Cartography  and  geodesic  products 

•  Friendly  issues 

-  Locations/number/availability  of  personnel 

-  Equipment/aircraft/vehicle  limitations  and  availability 
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Operations  Execution 


R 


•  Joint  Operations  Center  (JOC)  monitors  24/7 


-  Variety  of  live  feeds  inc.  Common  Operational  Picture,  variety  of  satellite  comm, 
nets,  live  streaming  video  (when  available),  weather,  chat 


-  During  mission  execution,  significantly  higher  manning 

•  Personnel  coordinate  across  comm.  Nets,  across  systems,  react  to  tactical 
situation 

-  Actions  could  include  retasking  personnel/assets,  coordination  with 
other/higher  Hq. 


EWall  News  view 

Following  2  slides  intended  to  represent  end  of  one  day 

and  beginning  of  next 
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liaciUJ 


Intel  Sequence 


NAV^AIR 


•  JSOTF  Analyst  sends 
card  to  three  separate 
analysts,  at  three 
different  organizations 
in  three  different 
locations,  all  working 
together  for  locating 
Saddam  Hussein 

•  Analyst  separately 
check  their  databases 
and  information, 
sending  information  to 
the  JSOTF 

•  JSOTF  analyst 
prepares  report  for  the 
commander 
recommending 
location  as  potential 
target 

•  CDR  agrees  and 
directs  information  to 
the  JPG  for  detailed 
planning 


JSOTF  Analyst 


A 


Interrogation  Report: 
Abid  al-Musslit 
29  Sep  03 


OGA  Analyst 


V  Corps  Analyst 


SOCOM  Analyst 


C 

PreHSTor 
Video 
0245Z 
30  Sep  03 


DIA  Imagery 
Tikrit  Oparea 
0127Z  Oct  03 


JSOTF  Analyst 

i 

JSOTF  Commander 


Joint  Planning  Group 
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Joint  Planning  Group 


Joint  Planning  Group  receives 
bundled  Intelligence 
Assessment  with  CDR’s 
direction  to  plan  mission 

JPG  Main  prepares  draft 
briefing  and  sends  it  to 
supporting  locations:  Special 
Forces,  3ID,  Aviation  Support 
Element,  CVGB,  etc 

Using  E-wall,  planners  provide 
real  time  collaboration  to  JPG 
Main,  providing  unit  level 
inputs,  final  product  is  briefing 
for  JSOTF  Commander 


Operation 

RED  DAWN 

13  December  2003 

Draft  Briefing 


Various  inputs  from  various  locations 
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NAV^AIR 


Final  Ops  Brief 


•  Intel  Analyst  found  initial  starting  point,  relayed  to  three  locations  for 
further  information  and  analysis  including  confidence  level  of 
information 

•  Intel  Analyst  prepared  recommendation  to  Commander, 

Commander  concurred  and  sent  information  to  JPG  for  detailed 
mission  planning 

•  JPG  conducted  collaborative  planning  and  put  together  Draft  briefing 
for  JSOTF  Commander 

•  He  approves,  directs  mission  and  JPG  passes  to  Joint  Operations 
Center  for  mission  execution 


Operation 
RED  DAWN 

13  December  2003 

JSOTF  Executes  Mission  while  JOC 

monitors 
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Ewall 

Special  Warfare  Scenario 
Demonstration 
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FY04  Accomplishments 


R 


O  Completed  CASC  phase  II  experiment  examining  asynchronous,  distributed  team 
collaboration  using  a  realistic  NEO  collaborative  scenario  within  a  Ewall  collaboration 
environment  resulting  in  a  better  understanding  of  the  cognitive  mechanisms  used  by  these 
teams 

H  Transitioned  CASC  Phase  II  data  sets  and  NEO  scenario  to  Peter  Foltz,  ACT  and  Jared 
Freeman,  Aptima  to  support  related  CKM  research.  Sending  data  sets  to  Nancy  Cooke,  ASU. 

O  Completed  Collaboration  Advisory  Tool  (CAT)  empirical  evaluation 
O  Papers  /  Presentations 

Warner,  N.  and  Wroblewski,  E.  The  Cognitive  Processes  used  in  Team  Collaboration 
during  Asynchronous,  Distributed  Decision  Making.  Paper  presented  at  the  Command 
and  Control  Research  and  Technology  Symposium,  San  Diego,  Ca.  June,  2004. 

Wroblewski,  E.  and  Warner,  N.  Decision  Support  and  Automation  Technology  to  Improve 
the  Warfighter’s  Tactical  and  Strategic  Decisions.  Presentation  at  the  Naval  Air  Systems 
Command  Technology  Showcase,  China  Lake,  Ca.  May  2004. 

Warner,  N.  and  Wroblewski,  E.  The  Process  of  Achieving  Collaborative  Knowledge  In 
Asynchronous  Collaboration.  Presentation  at  the  Collaboration  and  Knowledge  Management  Workshop, 
University  of  San  Diego,  San  Diego,  Ca.  January,  2004. 

Warner,  N.,  Letsky,  M.,  and  Cowen,  M.  Model  of  Team  Collaboration.  Informal  Peer  Review,  October,  200< 

£3  Updated  NAY  AIR  Collaboration  Testbed 

-  Current  Ewall  software  (Aug  2004) 

-  NEO  scenario  with  Ewall 

-  Ewall  demonstration  with  Special  Warfare  scenario 
o  Established  Transition  Vehicles  for  CKM  Products 


-  Special  Warfare  Command 

-  Joint  Forces  Command 
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Backup  Slides 
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CASC  Phase  II 

Definitions  for  Collaboration  Stages  * 


Knowledge  Construction:  (KC)  “Team  members  reading,  clarifying  information  and  understanding  the  problem.” 

1.  Individual  team  members  reading  information 

“  I  need  to  read  mine”  (reads  specific  environmental  information) 

2.  Clarifying  information 
“isn’t  the  port  guarded?” 

“the  rebel  forces  consists  of  500  trained  solders  (Stating  the  facts  without  applying  that  knowledge  to  any  possible  solution) 

3.  Understanding  the  problem 

“The  goal  is  to  save  the  people.”  (Realizing  what  they  have  to  accomplish.) 

Collaborative  Team  Problem  Solving:  (TPS)  “Team  members  communicating  data  and  knowledge  to  develop  solution  options  to  the  problem.” 
(solution  options  are  defined  for  each  of  the  five  components  of  the  final  plan  —  i.e.  personnel,  transportation,  weapons,  critical  times  and  detail  plan) 

1.  Analyzing  the  data  to  come  up  with  a  solution 

“if  they  fly  the  helo  high  enough  over  enemy  lines  that  gives  us  a  way  in  and  out” 

2.  Using  data  to  justify  a  solution 

“we  could  use  the  navy  seahawk  as  it  can  hold  up  to  1 1  extra  people;  the  7  man  team  the  navy  uses  plus  the  three  workers  that  is  ten  people  so  you  have 
space  on  the  helo  for  everybody” 

3.  Developing,  rationalizing  and  discussing  solution  alternatives 

“If  you  take  two  seahawks  over  there  that’s  four  machine  guns  and  plenty  of  missiles  for  A/A  and  A/G” 

4.  Establishing  a  plan  of  approach. 

“Let’s  think  it  through” 

5.  Convincing  others  of  a  specific  thought  without  absolute  consensus  of  the  final  solution 
“you  want  it  to  be  pretty  dark  by  the  time  you  get  to  the  church”  —  “Yeh” 

Team  Consensus:  (TC)  “Team  negotiation  of  solution  option  and  final  agreement  by  all  team  members  on  a  particular  option.”  (solution  option  can 
be  any  one  of  the  five  components  in  the  final  plan) 

1.  Agreeing  on  the  final  solution 

“So  we  should  go  straight  from  the  army  base?  -“Yeh”;  “Yeh” 

Outcome  Evaluation  &  Revision;  (OER)  “Team  evaluation  of  selected  solution  option  against  problem  solving  goal.  Team  revises  solution  option  if 
option  does  not  meet  goal.”  (solution  option  can  be  any  one  of  the  five  components  in  the  final  plan) 

1.  Choosing  to  accept  the  final  decision  or  revise  it 

“Use  the  C-130  to  parachute  army  rangers  into  a  mile  of  church;  drive  truck  to  church  &  pick  up  people  and  drive  one  mile  to  pick  up  point  by  blackhawk” 

—  CHANGE  SOLUTION  OPTION:  “drop  rangers  at  church  using  blackhawk  and  pick  up  people  using  hoist  in  a  second  blackhawk” 

Additional  Stages:  (MISC)  “Other  unique  team  behavior  not  described  in  the  above  categories.  Need  to  describe  unique  behavior  and  label  stage.” 

1.  Any  utterance  that  isn’t  applicable  to  any  other  collaboration  stage. 

*  =  confirmation  utterances  like  “yes”,  “yeh  ”,  “ alright ”  during  any  of  the  stages  should  be  associated  with  there  respective  utterance(s). 


CASC  Phase  II 
Cognitive  Process  Definitions 
(see  definitions  of  data,  information,  knowledge  and  wisdom.  Bellinger,  G.,  Castro, 
D.,  and  Mills,  A.  (2004),  http://www.systems-thinking.org/dikw/dikw.htm) 


1.  Metacognition  dti:  individual  conversion  of  data  to  information  =  individual  team  member 
converting  data  to  information. 

“I  noticed  the  rebels  don’t  have  night  vision  and  it’s  foggy  in  the 
morning  till  10  am  ( data );  that’s  to  our  advantage  ( information  —  no 
action )” 


2.  Macrocognition  imm:  individual  mental  model  construction  =  individual  team  member, 
using  available  information,  develops  his/her  mental  picture  of  problem  situation. 


“It’s  two  in  the  morning  in  Jan.  and  Jan.  is  the  rainy  season  for  this 
island;  the  island  is  750  miles  north  of  Australia  and  is  mostly  rain 
forest  and  vegetation;  can  only  get  battleships  within  a  mile  of  coral 
and  at  low  tide  can’t  get  a  single  man  raft  onto  the  island;  before 
noon  heavy  fog  and  by  noon  it  will  be  gone” 


3.  Macrocognition  itk:  individual  task  knowledge  development  =  individual  team  member 
asking  for 

clarification  to  data  or  information;  response  to  clarification. 

“isn’t  the  port  guarded?”(clarification) 

“Yes,  the  port  is  guarded”  (response  to  clarification  question) 

“How  do  we  get  around  the  rebels?”  (clarification  to  strategy 
question) 

“I  am  the  environmental  expert”  (clarifying  data) 

“We  have  5  minutes  remaining”  (clarifying  data) 

“yes,  the  seals  are  better  in  water”  (clarification  of  facts) 


4.  Macrocognition  tk:  team  knowledge  development  =  All  team  members  participate  in 
clarifying  (i.e.  answering  a  question)  information  to  build  team  knowledge. 

•  “Is  there  windows  in  the  church?”  (E)  —  information 

•  “Eight  windows  in  the  church”  (W)  —  information 

•  “Eight  windows  in  the  church”  (I)  —  information 

•  “The  only  problem  is  they  are  not  going  to  do  us  any  good  because 
they  (workers)  are  in  a  inside  room”  (I)  —  information 

•  “They  are  all  external  rooms,  they  (workers)  are  not  in  an  external 
room”  (I)  —  information 

5.  Macrocognition  ko:  knowledge  object  development  =  pictures,  icons  or  standard  text, 
developed  by  an  individual  team  member  or  the  whole  team,  that  represents  a  standard  meaning 
to  the  team. 


•  “weapons  expert,  type  our  responses  to  the  final  plan  on  the 
form” 

6.  Macrocognition  vrm:  individual  visualization  and  representation  of  meaning 

Visualization  =  individual  team  members  use  methods  (e.g.  graphs,  pictures) 
to  transfer  meaning  to  other  team  members 

•  “team  members  use  scenario  map(s)  to  exchange  distance 
information  between  objects  like  the  enterprise  and  the  island” 

Representation  =  individual  team  members  use  methods  to  sort  data  and 

information  into  meaningful  chunks 

•  “using  yellow  note  pads  to  sort  data  into  categories” 

7.  Metacognition  cu:  team  integration  of  individual  knowledge  for  common 
understanding  = 

all  team  members  combine  individual  pieces  of  knowledge  to  achieve  a  common 
understanding. 

“I  think  we  should  come  in  at  7:00am  because  fog  comes  in  at  6:00am 
and  it  will  help  us”(W) 

“I  agree  it’s  still  dark  &  rebels  have  no  NVG”  (I) 

“I  also  agree  because  its  critical  that  we  do  the  mission  covertly”  (E) 


8.  Macrocognition  kio:  knowledge  interoperability  development  =  team  members  exchanging 
knowledge  among  each  other. 

•  “we  need  to  take  them  out  of  the  church  before  noon”(derived 
knowledge  from  data  and  information) 

•  “I  don’t  think  the  SEALS  should  be  used  (negative  solution  with  no 
justification) 

•  “I  think  the  Army  is  best”  (no  justification) 

9.  Macrocognition  ica:  iterative  information  collection  and  analysis  =  collecting  and 
analyzing  information  to  come  up  with  a  solution  but  no  specific  solution  mentioned. 

•  “It  seems  we  have  better  NVG  capability  than  the  rebels”  (providing 
analyzed  information) 

•  “I  think  we  should  first  start  talking  about  our  expertise”  (i.e. 
approach  to  develop  a  solution) 

•  “Are  we  trying  to  go  up  on  the  roof?”  (i.e.  clarification  of  plan) 

10.  Macrocognition  tsu:  team  shared  understanding  development  =  discussion 
among  all  team  members  on  a  particular  topic  or  data  item  (i.e.  discussion  does  not 
involve  answering  questions) 

•  “No  we  have  no  translator  that  speaks  drapoize”  (I) 

•  “But  it  says  we  have  a  translator”  (W) 

•  “He  is  highly  skilled  in  many  different  languages”  (I) 

•  “Then  what  you  are  hoping  is  that  there  is  a  translator  on  the  island  that 
speaks  draponize  and  another  language  that  our  translator  also  speaks” 
(E) 

11.  Macrocognition  sa:  develop,  rationalize  and  visualize  solution  alternatives  =  using  data  to 
justify  a  solution 

•  “alright  if  we  are  looking  at  the  navy  we  could  use  the  navy  seahawk; 
it  can  hold  up  to  1 1  extra  people” 


Macrocognition  cmm:  convergence  of  individual  mental  models  to  team  mental  model  = 

convincing  other  team  members  to  accept  specific  data,  information  or  knowledge 

•  “That  sounds  really  good” 

•  “I  see  what  you  are  saying” 

•  “Yes”  (confirmation  by  an  individual) 


13.  Metacognition  cs:  team  agreement  on  a  common  solution  =  all  team  members  agree  on  the 
final  plan. 

•  “1  agree  with  the  final  plan”  (W) 

•  “1  agree  also”  (1) 

•  “Yes”  (E) 


14.  Macrocognition  tn:  team  negotiation  of  solution  alternatives  =  team  negotiation  of 
solution  alternatives  ending  in  a  final  solution  option,  (solution  options  are  defined  for  each  of 
the  five  components  of  the  final  plan  —  i.e.  personnel,  transportation,  weapons,  critical 
times  and  detail  plan) 

Weapons  Example: 

•  “F-18  for  backup  cover’  (E) 

•  “(F-18)  on  standby”  (W) 

•  “Yes”  (E) 

•  “Used  to  take  out  oncoming  enemy  threat”  (I) 

15.  Macrocognition  tpr:  team  pattern  recognition  =  the  team  as  a  whole  identifies  a  pattern  of 
data,  information  or  knowledge^ 

•  “It  seems  like  all  the  needed  assets  (SEALS,  Blackhawk,  F-lS’s) 
are  located  on  the  USS  enterprise”(E)  —  needed  assets  are  the 
patterned  information. 

“Yes”(I) 

Yes  (W) 


16.  Macrocognition  ct:  critical  thinking  =  Team  working  together  toward  a  common  goal, 
whereby  goal  accomplishment  requires  an  active  exchange  of  ideas,  self-regulatory  judgment,! 
and  systematic  consideration  of  evidence,  counterevidence,  and  context,  in  an  environment  where 
judgments  are  made  under  uncertainty,  and  there  is  limited  knowledge  and  time  (Hess  & 

Freeman,  2004). 

1.  critical  thinking  is  measured  as  a  composite  of:  (Warner  &  Wroblewski,  2004; 
Hess  &  Freeman,  2004) 

•  MCitk:  individual  task  knowledge  development  =  individual  team 
member  clarifying  data;  asking  for  clarification. 

>  “isn’t  the  port  guarded?” 

•  MetCcu:  team  integration  of  individual  knowledge  for  common 
understanding  =  one  or  more  team  members  combine  individual  pieces 

of  knowledge  to  achieve  a  common  understanding. 

y  “We  need  to  figure  out  the  critical  time  of  when  we  can  leave  by 
each  of  us  providing  our  leave  times” 

•  MCkio:  knowledge  interoperability  =  team  members  exchanging 
knowledge 

among  each  other. 

>  “when  we  take  them  out  it’s  got  to  be  at  night”  (derived  knowledge 
from  data) 

•  MCsa:  develop,  rationalize  and  visualize  solution  alternatives  =  using 
data  to 

justify  a  solution 

>  “alright  if  we  are  looking  at  the  navy  we  could  use  the  navy 
seahawk;  it  can  hold  up  to  1 1  extra  people” 


Note:  one  critical  thinking  frequency  count  =  oneMCitk  +oneMetCcu  +  MCkio  + 
MCsa 


17.  Macrocognition  shk:  sharing  hidden  knowledge  =  individual  team  members  sharing  their 
knowledge 

through  prompting  by  other  team  member(s). 

•  “When  does  the  sunrise?”  (W) 

•  “Isn’t  that  your  area  of  expertise”  (I) 

•  “Ok,  sunrises  at  6:00am”  (E) 

18.  Metacognition  sag:  solution  adjustment  against  goal  and  exit  criteria  =  team  as  a  whole 
compares 

complete  solution  option  (i.e.  all  five  components  of  final  plan)  against  goal  and  exit  criteria. 

•  “  Our  solution  options  meet  the  24  hr  rescue  requirement  with  minimum 
enemy 

contact”  (I) 

•  “Yes”  (W) 

•  “Yes”  (E) 

19.  Macrocognition  csg:  compare  solution  options 

solution 

options  (i.e.  any  of  the  five  solution  components) 
cross 

workers  within  24  hrs). 

•  “  The  Navy  SEALS  and  the  Blackhawk  will  meet  the  goal  but  our  detail 
plan  cannot  be  completed  in  24  hrs”  (W) 

•  “We  need  to  re-look  at  the  detail  plan”  (I) 

20.  Macrocognition  aro:  analyze,  revise  solution  options  =  team  members  analyze  final 
solution  options 

(i.e.  any  of  the  five  solution  components)  and  revise  if  necessary. 


against  goal(s)  =  team  members  discuss 
against  the  scenario  goal  (i.e.  rescue  3  red 


“We  may  need  to  re-consider  using  a  window  to  get  into  the  church” 


